2 links tagged with all of: web crawling + mcp + ai agents
Click any tag below to further narrow down your results
Links
Firecrawl is an open-source toolkit that converts websites into clean markdown, with a local binary, a hosted cloud service and a self-hostable server. The repo README explains how to install the local version, plug it into AI tools as an MCP server, and call the same API from existing SDKs.
- Firecrawl Local runs as one binary from crates.io, scraping, crawling and mapping sites without a database, queue or container, and it works with Claude Code, Cursor, Windsurf and similar tools via MCP.
- Pages that need JavaScript render in a Chrome or Chromium already on your machine, and the local server answers the Firecrawl v2 API on port 3002 so existing SDK code runs unchanged.
- Firecrawl Cloud adds search, structured JSON, interact and agent features, plus access to 550+ data providers through Alexandria, with a free tier of 1,000 credits a month.
AgentReady is an open specification laying out what a website should do so AI agents can find, read, and act on it. The research behind it, drawn from over 250,000 agent runs, suggests that sites agents can't read get answered from other people's pages instead.
- Sites agents couldn't read were 3.7 times as likely to produce answers missing the facts the user asked for, and agent-ready businesses were clearly recommended about 1.9 times as often.
- In the traced runs, 82% of final answers traced back to a page the agent fetched, and docs pages were the most common source at 47%.
- Blocking is the big failure: not-agent-ready sites blocked agents 2.1 times as often, and a WAF returning a 403 or challenge page stops agents regardless of robots.txt.